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ABSTRACT 

In response io the specifications of a United States 
Office of Education grant "Cc.mmon Core of Data for the Seventies" 
(CCD-70), the Michigan Department of Education G. eloped a procedure 
for identifying user's requirements relative to & ;5tate Education 
Information System. The project sought to identify users, to locate 
and prioritize their management concerns, to identify related linkage 
questions, and to relate these to the flow of Information. The tools 
developed by the study included an interview guide and a consensus 
survey employing a modified Delphi Technique. Results indicated that 
these procedures responded tc users' needs, for thoy generated the 
major questions faced by users, identified the data needed to answer 
these questions, and located the data which was available. It was 
concluded that these procedures could be adopted in other contexts 
since they represented a viable means for identifying user needs with 
respect to management information systems, (Author) 



ERIC 



A PROCEDURE FOR IDENTIFYING 
USER'S REQUIitRMENTS FOR A 
MANAGEMENT INFORMATION SYSTEM 



0 4 OePA<^TM6NT Of MCaLTH- 
EDUCATION * WEL^ARf 
NATIONAL INSTITUI t OP 
I DUCAT ION 

TmiS OOCuVfNr HAi PEES repro 
OuCEO tX»CUY AS HECCfVED '«0V 
The PE ffiO^Off CIOANiJA'iON OmC N 

ATbNOiT poiNTSOF vnE A 0*; Opinions 

STAtEO DO %07 NECESSARav fiERRE 
SENT O^s CiAi. NA1iO%Al iNiTiTyTE Of" 
EDUCATION POii^iON OH POLICY 



Stanley A. Ruinbaugh 
and 

Dorothy R, VanLooy 
Michigan Department of Education 



The research reported herein was performed pursuant 
to a grant with the Office of Education, U,S, De- 
partment of Health, Education, and Welfare. Con- 
tractors undertaking such projects under Government 
sponsorship are encouraged to express freely their 
professional judgment In the conduct of the project. 
Points of view or opinions stated do not, therefore, 
necessarily represent official Office of Education 
position or policy. 



Presented at 
The 1974 Annual Meeting 
of 

The American Educational Research Association 
Chicago, Illinois 
April 15-19, 1974 



INTRODUCTION 



In a time of ever-Increasing demands at federal » state » and local levels 
for more and better Information about educatloni there Is a need, for a systematic 
and economical means of collecting, managingi and disseminating educational data* 
There are two major forces which create compelling reasons for a comprehensive 
information system in the Michigan Department of Education. Both of these forces 
are related to the broad concept of accountability. The first revolves around 
meeting the educational needs of children and youth in Michigan in a systematic 
and economical way; the t»econd derives its impetus from the impact of planning 
by all State Governmental Agencies for implementation of a Program Budgeting 
and Evaluation System (PBES). 

In order to implement a comprehensive information system, a solid data 
base is needed from which to address policy and management questions. In 
Michigan, the requirements of data users are diverse, but at the same time, they 
are related to pertinent and mutual issues faced by decision^makers. 

As an initial step in the development of a system-base, it is essential 
to identify the requirements of the users of data. It is to this task that 
this paper is addressed. 

In response to the specifications of a United S ites Office of Education Rvant 
"Common Core of Data for the Seventies" (CCD-70) , The Michigan Department of 
Education developed and tested a procedure for identifying users* requirements * 
for the financial module o£ a State Education Information System. It will be 
the purpose of this paper to describe the procedures developed for identifying 
users requirements, and to briefly describe how these procedures were applied, 
without going into the details of the Michigan study. The methodology that 
was developed should be equally applicable to all modules of a management 
information system. 
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The theory behind the Michigan Department of Education's study to 
Identify users* needs was that the Identification of the questions to which 
the data must respond Is the first step to be undertaken. If the major manage- 
ment and policy questions of the state's educators can be identified i it will be 
possible to identify their information requirements and to identify, collect, 
analyze I and usj3 data needed to meet these requirements* y 

The specific objectives of Michigan's CCD-70 grant were to identify and 
prioritize major management and policy questions, identify the related linkage 
questions (specific, data related questions) and data base, and relate these 
results to the existing information flow. 

It was assumed that more specific questions would demand specific data 
items while more general questions would require more generally applicable data 
items or, perhaps more likely, combinations of data items from a common data 
base. The relationships between the major and/or linkage questions nay be 
shown as an interlocking series of pyramids (see Figure 1) wHh the shaded areas 
represent ing^^the common core of data to be used to answer overlapping questions. 



MAJOR aUESTIONS 



MAJOR QUESTIONS 



MAiOR QUESTIONS 




OMMON CORE OP DATA- 



DATA BASE 

FIGURE 1: interconnecting Relationship of Data Elements to Major 
and Linkage Questiona. (Shaded Areas Repreaeixt Common 
Core of Data) 
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Flgure 1 indicates the following: (1) some linkage questions may share 
no common data elements; (2) some linkage questions will share common data 
elements,' and (3) all major questions are likely to share some common data 
elements. In order to Identify the major management dnd policy questions, 
linkage questions anc: required data» a personal interview approach was used, ,^ 
The individual tasks required to meet these objectives were as follows; 

1) Develop an interview guide 

2) Test the interview guide 

3) Conduct Interviews 

4) Analyze the interview results 

5) Identify major management and policy questions 

6) Identify the subset of major questions with particularly strong , 
fiscal implications 

7) Develop a method to prioritize these questions 

8) Prioritize the major questions 

9) Relate major questions to linkage questions and data requirements 

10) Identify financial Information that is presently available 

11) Identify areas of greatest existing need 

The completion of these tasks has resulted in '*The Identification of Users ^ Require-- 
uients for a State Education Information System.'** 

The following sections of this paper present discussion on 

1. The interview process 

2* Construction of data sets 

3. Analysis of results » and 

4. Conclusion 



*The study identified users' requirements for the financial module of a State 
Education Information System. However, the procedures reported herein should 
be equally applicable to each module of an information system. 

o 
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THE INTERVIEW PROCESS 



The interview process involved three basic fiteps: (1) development of the in- 
terview guide I (2) the selection of people to bi interviewed and (3) conducting of 
the interviews. Each of these steps required cqnsiderable time and planning in order 
to respond to the requiretnents of the specific study and at the same time to main- 
tain the potential for broader applicability. 

The Interview guide developed for Michigan's CCD-70 project has a personal 
data sheet and five sections, (See Attachment A). Section I collects the major 
management and policy questions and linkage questions. In Section II the Inter- 
viewee is asked to indicate the information that he hmt or would like to have in 
order to answer each question in Section I. Section III is designed to establish 
an overview of data needs. The questioning in this section is specifically 
directed toward the respondent's need for financial data. A matrix with funds, 
allocation and expenditures as variables on one axis, and federal, state and 
local levels on the other serves as the framework or outline for this group of 
questions. Section IV asks questions about what financial data the respondent 
uses, what he has, and what he would like to have. The final section, Section V, 
requests unstructured comments and recommendations from each person being 
interviewed. 

The intent in choosing the people to be interviewed was to obtain a representa- 
tive sample of management people in the K-12 sectov of the State Department of Educa- 
tion. The final trelection Included: The Deputy Superintendent; three Associate 
Superintendents; one member of the State Board of Education; seven Service Area 
Directors; seven Progx-am Administrators ; five Coordinator^ of Federal Programs; 
and personnel from two Local Education Agencies (6 individuals). The following 



-5- 

organizational chart (Diagram 1) Indicates the relationship of the sample 
selection to the Departmental structure* 

The selection of two Local Education Agencies (LEA's) can» at best, be 
considered only as an Indicator of local data needs* It was not possible to 
obtain a representative sample of local systems In the time frame allowed. 
Instead, staff of a rather large (23,761 students) system and a fairly small 
(3,787 students) system were Interviewed. One day was spent In each system. 
In each case the superintendent, business official and curriculum director were 
Interviewed. Their Inputs were considered as Indicators of the slmularlty or 
diversity between their data needs and those of the State Education Agency (SEA)* 

The Interviews followed a basic format. Each Interviewee received a memo Intro- 
ducing the staff and Indicating that he would be contacted for an appointment for 
the Interview. The Interviewer was accompanied by a stenographer who recorded 
the Interviewee's comments. In this way, the person being Interviewed could 
respond at a rate that was natural for him and there would be a transcript of 
the Interview to fill In any omissions. Each Interview was assigned a codo 
number and cannot be associated with the name of the person Interviewed, except 
by the Project Director. The Interviews ranged In time from one to four hours 
and were essentially uninterrupted sessions. For the most part, the people 
Interviewed weie very enthusiastic about the project and eager to participate. 
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CONSTRUCTION OF DATA SETS 

A reasonably accurate overview of an organization's Information needs 

can be established with four data sets:* 

Data Set I The major matragement and policy questions 

Data Set II A prioritized subset of major questions relating to th<* 

information module being developed (in this case financial) 

Data Set III The data required to respond to the questions in Data Set II 

Data Set IV The data presently available within the module under 
consideration* 

The first three of these Data Sets can, for the most partr'b^e constructed 
from a rigorous analysis of the interviews. Data Si\t II requires the application 
of a prioritizing process* The manner in which Data Set IV is constructed, 
would depend on data control procedures within the organization. 

Following is a brief description of the four Data Sets from the Michigan 
study and how they were constructed* 
Data Set I - Major Questions 

The first data set resulting from careful analysis of the transcripts of 
the interviews consisted of 111 major questions identified in 30 interviews. 
In each interview, the interviewee was asked to identify the management and 
policy questions or concerns that were of primary importance to him. These 
broad scopes questions will be referred to as major questions* Each person 
was also asked to supply the more detailed and specific questions that would 
need to be answered in order to deal with the major questions; these more 
specific questions are referred to as linkage questions. 
Data Set II - Prioritized Fiscal Questions 

The large number of major questions made prioritization of all of them 
unmanageable. Therefore, in keeping with the objectives of the grant, the 
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*The data sets are available upon request. 
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questlons with fiscal Implications were Identified for prioritization. The 
questions with the most prominent fiscal implications were subjectively selected 
by the researcher. This subset of thirty major questions was chosen for priori- 
tization, in order to identify the priority of major Issues. Data Set II con- 
talned the thirty major fiscal questions with the 153 associated linkage questions, 
ordered according to the results of the prioritization process; and graphic 
reoresentatlryns of the rp.«ii.lt« from GAch of the steps in the process (Attachment B) 
Only the stAte level manaaetnent People were involved in the prioritization 
process . 

The prioritization process employed a modified Delphi technique. The Delphi 
technique was developed by the Rand Corporation and was modified for this applica- 
tion, under contract, by Person-O-Metrlcs, Inc. The Delphi technique is a fore- 
casting and consensus technique for anonymous contributions from expert or con- 
cerned opinion. Delphi questionnaires seek a group Judgment by offering a con- 
tinuum of responses » providing controlled feedback on previous group and 
individual judgments, and forcing the range of judgments to converge. Opinions 
of those not in agreement with the majority are recorded and used as feedback to the 
group. A statistical response of the median and interquai-tile range (cilddle 50%) 
is usually reported. 

The consensus survey, or prioritization of the major questions, was performed 
in three phases. The technique used in each of the three phases is discussed in 
the following paragraphs. Each phase involved the sending of materials solid- 
ting responses from each person who had been interviewed. The respondents wer^ 
given a three to four day time span to return the materials. An analysis of the 
results was performed and the next phase prepared for distribution within a week. 

The first phase of the survey was a **Q-sort" which simply involved the linear 
ranking of the thirty major questions. Each participant was given 30 cards with 
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one question on each card* Instructions Informed the respondents to divide the 
deck Into three aporoxlmately equal plies of *'very hlgh,^' and "moderately high 
to low" priorities for the entire Michigan Department of Education. The plies 
:^ere sorted again, merged, and numbered In rank order. 

The Q-8ort technique produced an ordinal ranking which was without the usual 
omissions and ties which tend to haunt many rank order exercises* The dis- 
tributions of opinions on the item questions were quite wide with very few item 
questions able to claim statistically significant differences over the overlapping 
distributions of their neighboring items. The top 16 items were maintained for 
the second round of the survey. The second phase of the consensus survey re- 
ported back to each respondent how he rated each of the sixteen items in comparison 
to the middle fifty percent of respondents. The respondent: was then requested 
to rate the item within the fifty percentile range, or give his reasons for 
not being willing to do so. An important variation in the usual Delphi technique 
was present in that the repositioning of the response was not only made in 
relationship to the group feedback but also in relation to the rank ordered 
responses to the items immediately above and below the item being repositioned. 
The results of the second phase were then calculated to evaluate the new inter- 
quartile range for each item. 

The results of the first Delphi round (the second round of the survey) contracted 
the Item distributions into sll^fty^smaller interquartile bands. Although the 
distributions did not approach convergence, the contraction was sufficient to 
allow most of the items to claim statistical 3lgnlflcance in their dlffe ances 
among the items more than one or two positions above and below. Sixty-five per- 
cent of the participants chose to state one or more reasons for not conforming 
to majority opinion* Each of the 16 items had at least one defender or critic 
arguing for a higher or lower priority* In the third and final phase, the respondent 
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was again asked to reevaluate his response for the six highest ranking issues. 
However^ in this round he was able to see the comments of his fellow re«pondents 
who wished to remain outside the fifty percentile range before making a final 
evaluation of his position. The results of this final round were accepted as 
the consensus of opinion on the priority of major issues. 

The statistical relationship among the top six items indicate the statistically 
significant (i»e., not likely due to chance) greater than (>) and approximately 
equal (^) comparisons between each item and the other five items, 

TOTAL OROUP 

BA Long range planning 

^ CE Equal educational opportunities 

> > CB Most effectively deliver services 

> > > BB Utilize human and financial redources 

> > > « AB Beet delivery system 

> > a > « AC Priority of programs 

The data were broken down by job and role classification. One groui> 
comprised of Service Area Directors and Executive Office was coapared to 
tho other participants consisting of Program Supervisors and Coordinators, 
The Service Area Directors and Executive Office spread their responses widely 
over the six items so that only tvo significant item differences could be^ 
proven with this small subsample. 

8KRVICE A«KA DIRBCTOB AHD EXECUTIVE OFFICE 

Long range planning 
^ Most effectively deliver services 

Utilize human and financial resoxirces 



BB 



« 



A^ Priority of programs 



tt » fir CE 



Equal educational opportunities 
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The Program Supervisors and Coordinators made up of an equally small 

> 

dubaampie aa that of the Directors and Executive Office were in greater 
agreement over ranking the Items* The^ltera on equal educational opportunities 
led the ranking with the priority of progratris question at the bottom. 

PftOORAJf'gt/PmiSORS AKD COORDINATORS 
CE Equftl educational opportunities 

^ BA Long range planning 

> ^ «BB Utilize human and financial resources 

> « ^ CB Most effectively deliver services 

> > « « AB Best delivery system 

> > > I* > AC Priority of programs 

On an item to item comparlsovi of the top six priorities » between the 
Directors and Executive Office Group and the Progracv Supervisors and Coordinators 
Group, the Directors and Executive Office showed the Priority of Programs item 
significantly higher in its distribution than in the Supervisors and Coordinators 
Group. The Program Supervisors and Coordinators Group were significantly 
higher in their positioning of the Equal Educational Opportunities item. 

All of the statistical significance testing used the Kologorov-Smirnov two- 
sample, one-tailed test. This is a powerful statistical procedure for small 
samples where the assumptions of a normal distribution (bell-shaped curve) cannot 
be made* 

Data Set III - Financial Data Needed 

The third data set, identification of financial data needed by management 
people within the State Department of Education, was organized by data source. 
In accordance with the Interview guide, each interview had four major areas that 
identified data needs. All data it^ms mentioned by the interviewee in one of 

these areas were listed in financial and non-financial data sets* The items in the 

i 
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financial data aete from each interview were merged and regrouped according to 
data source. Data Set III represented the indicated data needs, according to 
data source, of the people Interviewed. The data items were coded to Indicate 
whether or not they were available* Also, a code was used to Indicate to which 
of the major questions the data items responded. 
Data Set IV ^ Financial Data Presently Collected 

H The fourth and final data set is the financial data collected from the local ' 
and intermediate school districts. The Michigan Departmeilt^ Education's 
Research Data Program maintains up-to-date files of all collection documents 
that are sent out to the local and intermediate school districts. A thorough 
search of these files for documents requesting financial information, resulted 
In the information presented in Data Set IV. 

Indicated with the name of each collection document was the form number, respon- 
dent, collection date, package, service area requesting the information, data 
processing status and comments* These pieces of information are self-explanatory, 
except perhaps ''package/' In an attempt to relieve the responding burden of the 
local and intermediate agencies, the Department of Education has established 
collection packages that group collection documents with comparable functions or 
collection deadlines into a single package* There are presently three collection 
packages: Fourth Friday (information collected the fourth Friday after Labor 
Day), End of School Year, and Close of Fiscal Year. 
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Analysls of Results 

The analysis of the results of the four data sets is a very straight forward 
process. The major questions of the management staff | the data needed to answer 
those questions and the data available to answer thew> have been identifier!; in 
other words, the users' requirements have been identified* The information avail- 
able makes it possible to study patterns of information flow, areas with priority 
Information needs, information availability by management level and many other 
aspects of information exchange. Beyond the objective of identifying users' 
requirements, the broad scope view made available by the four data sets provides 
a frame work for systems design efforts. 

The purpose of the was fulfilled with the Identification of informa- 

tion that would be required In the data base for the financial module of a MIS« 
The type of information gathered for the study made it possible to carry out 
a more complete analysis that would be more specific and meaningful within the 
State Department of Education, The analysis of the results of the Michigan 
study Included a discussion of infonaation flow from each of the levels of the 
educational hierarchy (local, intermediate, state, and federal) to management 
people within the State Department of Education. Primary areas of data need 
were identified at each level. The data needs that were not being fulfilled 
at the time of the study were explored and discussed in light of existing in- 
formation flow; and recommendations for Improved information flow were associated 
with area of need* ' 



r Conclusion 

The methodology described above for Identifying users' requirements of a 
management information system was developed with a commitment to the premise 
that the identification of the users* questions must precede the identification 
of the users' data requirements. Also basic to this methodology is the belief 
that modular information needs can not be fully understood unless they are 
viewed in relation to a total information system. 

It is necessary to have an awareness of how the information in the module 
under consideration ^elates to the information that would be in the other modules 
of the systemi so that the capability for cross referencing can be anticipated 
and planned. Additionally, it is necessary to understand the implications of 
collection, storage and use of the information at each level of the system. 

The application of this Methodology in the "CCD-70" study, proved to be 
effective. Indeed, users requirements for the financial module of a management 
information system were identified. Also, recommendations, were made to improve 
information availability and flow. The potential for applying the methodology 
to other modules and for interrelating them and carrying them through to 
Implementation has begun to be explored, through additional studies at the 
Michigan Department of Education, and the prospects for continuing success 
are promising. 



ATTACHMENT A - Interview Guide 

Section I 
Section II 
Section III 
Section XV 
Section V 
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INTERVIEW GUIDE 

PERSONAL IMFORMATION 

i 

1. NAMBt ' ' 

2* Roott ^Bldg. P hon e S ervice Area 

3. ADHINIsmxIVB AREA: (Evaluation, Aseesroent, 71tle Itl). 

4. LEVEL OP RESPONSIBtLlTY 

Superintendent 
AsBociate Superintendent 
Director 

Program Supervialon 
Coordinator of Federal Program 
School District 

S» LEVEL OF PROGRAU RESPONSIBILITY (express in % time spent » 

if necessary) 

Federa l 

State 

Local_ 

Other 

IKTERVIBW mmBER 



INTERVIEWER 



mBRVlBW GUIDE 

Section I 

A0 A normal function of your po8ition» you ikiiat make decisions 
and have questions answered dally. I would like to establish a 
hierarchy of these questions; or to put it another Way» to have a 
listing of these questions and the major management or polity decisions that 
th<^ help you to make* (As an example, consider a classroom teacher; a 
major management type decision that the teacher might be concerned with is 
"How can I provide the best education for my students?" Having answers 
to the following questions would help to answer this question: 

1» Whiit are the students deficiencies? 

2. Where are they showing the greatest interest and 
responsiveness? 

3* What will be most useful to them?)» 
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Section III 

rms 

Sourci. 



ALLOCATION 
Recipient 



SPENDING 
Expenditure 




Federei 



State 



District 



School 
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state 



District 



School 



State 



District 



School 



State 



District 



School 



State 



District 



School 



SECtlOH IV 



r 



IHTERVIEW CUIDB 

What financial Information do you presently use the noflt? i.e. 
ehartil, tables, data ite«a (pleaee list, beginning with the .tost used 
information) . 



What financial infooMtion would you be able to make use of if 
it wtre available? (Please list and rank as above.) 



I 



Ik) you have need of any up-to-date infonation? (Please specify the 
type of ittlq|i»tion fad ^ «tequeacy» l.e; daily, we|kly, ponthly, seal- 
soaually.) 



IKTBRVIEW GUIDB 



Would your quedtiona he answered mora rapidly and more thoroughly 
by data that had undergone nanipul(\tion0 specified by you? (Calculationi 
reorganisation 9 etc.) 

i 



Section V 

Considering that our objectives are to supply the most needed 
financial data in the moat useful form, can you make any further suggestions 
that might facilitate our efforts? 



ATTACHMENT B 
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